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Abstract. This paper proposes a new method for vector quantization 
by minimizing the Kullback-Leibler Divergence between the class label 
distributions over the quantization inputs, which are original vectors, 
and the output, which is the quantization subsets of the vector set. In 
this way, the vector quantization output can keep as much information 
of the class label as possible. An objective function is constructed and we 
also developed an iterative algorithm to minimize it. The new method is 
evaluated on bag-of-features based image classification problem. 
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1 Introduction 


Vector quantization is a problem to quantize the continuous signal vectors into 
a discrete dictionary 13213715154134131] . The vectors are usually quantized into 
several discrete quantization subsets, and then the index of the subset will be 
used as its new representation [mu- It should compress the original vectors, 
and also be helpful for the effective representation of these vectors [17]. An 
example is the bag-of-features based image representation (18123l5li2lil5l38] . In 
this problem, the local image features are quantized to a visual dictionary. The 
quantization is usually conducted in an unsupervised way, assuming that the 
class labels are not available [35]. But it’s better to learn it by combining the 
class labels in a supervised way. The ideal quantization we are seeking is the one 
that can contains all the information needed for the classification I16I50I1I42I . 

In this paper, we present a new quantization algorithm so that the class 
label can be used. The algorithm is developed by first modeling the distribution 
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of the class labels over the quantization input (the original vectors), and the 
distribution of the class labels over the quantization output (the quantization 
subsets), and then minimizing the Kullback-Leibler divergence between the two 
distributions 1331711 212911 0141311191111 . We build an objective function to this 
end, which involvesthe quantization of the vectors. This idea is shown ni Fig. [0 
Our method is motivated by Wang et al.’work [3S], which learned the codebook 
for vector quantization in an supervised way. However, the differences are of 
tow-folds: 

— Wang et al. [3S] used the class label to define the margin and then maximize 
it, while we directly minimize the Kullback-Leibler divergence between the 
class label distributions over the quantization input and output. 

— Wang et al. [.'19| implement the vector quantization by using a codebook, 
while we directly quantize the vectors into subsets without using a codebook. 
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Fig. 1. Minimizing the distribution of class labels over the original vectors and the 
quantization subsets. 


The rest part of this paper is organized as follows: Section [2] introduce the 
proposed algorithm in details. Section [3] shows the evaluation of the proposed 
algorithm bag-of-features image classification. Finally, Section 6 concludes the 
paper. 

2 Method 

We first introduce the distribution of the class label over the vectors and the 
quantization subsets in Section 12.11 and then given the algorithm to minimis the 
Kullback-Leibler divergence between these distributions to quantize the vectors 
in Section [?~?1 
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2.1 Class label distributions 

Assume we have a set of N labeled training data samples, denoted as { (x, : , y i )}^L 1 . 
x,; £ R d is the d-dimensional vector of the i-th sample, and t/i £ y is its corre¬ 
sponding class label. The classification problem is to predict the class label of a 
sample from its vector. To estimate the class label distribution over the vector, 
we define the p(y |x) as the conditional distribution of class label y over a sample 
vector x, where y £y, and x £ {x^jA 1 . 


Class label distribution over input Given a input sample x, and a class 
label y, to estimate p{y |x, ; ), we first find its nearest neighbors Mi |f)l8ll20l40l36j . 


and then 


where 


J\fi = k — armin Xj | |x, — Xj \ 


p{y\ x i) = T^rr E 




J (yj = y) 


i, if yi = y 

0, else 


( 1 ) 

( 2 ) 

(3) 


Class label distribution over output Then we quantize the vectors {x;}^ 
into M non-overlapping quantization subsets, denoted as {<S m }^f =1 , as shown in 
Fig. [2] Given subset S rn , and a class label y £ y, the conditional distribution 
for y over S m is defined as 


p{y\s m ) 


1 


E 

XjGSm 


I (yj = y) 


(4) 


Kullback—Leibler divergence The Kullback- Leibler divergence between 
the two distributions p(y\S m ) and y(y|x,) are defined as ((5}). 


M 


D (p{y\x)\\p(y\S)) = E 1 E 


y&y fm=l 




(5) 


We try to have a quantization output {<S m }^ =1 from {x^jA 1 so that the the 
quantization can minimize the Kullback—Leibler divergence 


min 
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To solve this problem, we adapted a iterative algorithm. 


(6) 
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Fig. 2. Vector quantization from vectors to quantization subsets. 


2.2 Interactive algorithm 

In this algorithm, we repeat a two-step procedure for many times: 

— In this first step, we fixe the quantization subset as S° ld , • • • , and com¬ 
pute the class label distribution over the quantization output as: 

P”(V|S„) = ji E '(«=») (7) 

171 Xj£<s^ d 

— In the second step, we fixe the distribution and update quantization subset 
by ©. 
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When a new sample x not in the training set is given, it is quantized to the 
quantization subset as 


where 


Sm* t X 


TO 


argmin 

m =,M 



p(y l x ) \ 
new {y\S m )J 


(9) 


( 10 ) 


3 Experiment 

Experimental evaluation is given in this section on a real data set with a bag-of- 
features based image classification problem. We use the Fifteen Natural Scene 
Categories database in this experiment [18]. There are 15 classes and for each 
class, there are around 200 or 300 images. We use the proposed vector quanti¬ 
zation algorithm to represent the image as a quantization histogram under the 
bag-of-features framework. 

Fig.Q]shows the results for the Fifteen Natural Scene Categories data set. The 
kmeans algorithm is used as a baseline quantization method I6I45I3I25I55I441 . As 
in Fig. [I], the proposed method outperforms the the baseline method. 


4 Conclusion 

In this paper, the problem of vector quantization is investigated. The idea is 
motivated by the supervised quantization dictionary learning method proposed 
by Wang et al. [39]. We try to keep all information about the class label from 
the quantization input to the output. It is implemented by minimizing Kullback- 
Leibler Divergence between the class label distributions over quantization input 
and output. This method can also be applied to social media data analysis 
1211571741 . user recognition of mobile m, transportation prediction , web 

news extraction [43] . malicious websites detection 47l49l30l52l4fil48l53l . 
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